Spatial Information Guided Convolution for Real-Time RGBD Semantic Segmentation

3D spatial information is known to be beneficial the semantic segmentation task. Most existing methods take data as an additional input, leading a two-stream network that processes RGB and separately. This solution greatly increases inference time severely limits its scope for real-time applications. To solve this problem, we propose Spatial guided Convolution (S-Conv), which allows efficient feature integration. S-Conv competent infer sampling offset of convolution kernel by information, helping convolutional layer adjust receptive field adapt geometric transformations. also incorporates into learning process generating spatially adaptive weights. The capability perceiving geometry largely enhanced without much affecting amount parameters computational cost. We further embed network, called Guided Network (SGNet), resulting in state-of-the-art performance on NYUDv2 SUNRGBD datasets.